A web crawler can browse and search a large number of web sites that contain publicly available data and then create a copy of the public web pages for later processing by a search engine to index the copied web pages. When a user makes a request to search the web, the web search engine does not search the real web sites, but rather searches the indexed copies of the web pages and returns the search hits from the indexed copies. A web search is not a real-time search and only provides publicly available data on the web.